System architecture for very fast ethernet blade

ABSTRACT

The system of the present invention provides data transmission speeds at or in excess of 10 gigabits per second between one or more source devices and one or more destination devices. The system comprises a media access control (MAC) interface to facilitate receipt and transmission of packets over a physical interface. A first field programmable gate array is coupled to the MAC interface and operative to receive packets from the MAC interface and configured to perform initial processing of packets, which are dispatched to a first memory. A second field programmable gate array is operative to retrieve packets from the first memory and configured to compute an appropriate destination, which is used to dispatch packets to a backplane. A third field programmable gate array is provided that is operative to receive packets from the backplane and configured to organize the packets for transmission, which are dispatched to a second memory. A fourth field programmable gate array is coupled to the MAC interface and operative to retrieve packets from the second memory and configured to schedule the transmission of packets to the MAC interface for transmission to one or more destination devices.

CROSS-REFERENCE TO RELATED APPLICATION

This application is a continuation of and claims priority under 35U.S.C. §120 to U.S. patent application Ser. No. 10/139,831, entitled“SYSTEM ARCHITECTURE FOR VERY FAST ETHERNET BLADE,” filed May 6, 2002,now U.S. Pat. No. 7,266,117, issued Sep. 4, 2007, assigned to the sameassignee as the present application, and incorporated herein byreference in its entirety.

COPYRIGHT NOTICE

A portion of the disclosure of this patent document contains materialwhich is subject to copyright protection. The copyright owner has noobjection to the facsimile reproduction by anyone of the patent documentor the patent disclosure, as it appears in the Patent and TrademarkOffice patent files or records, but otherwise reserves all copyrightrights whatsoever.

BACKGROUND OF THE INVENTION

The invention described herein relates to computer networking and, inparticular, to improved methods, systems, and software for routing dataat high speeds through a switch or other network routing device.

The explosive growth of the Internet has brought more and more usersonline every day, and computer networks have assumed an increasinglyimportant role in today's highly interconnected world. As usersincreasingly rely on the network to deliver required data, networktraffic has increased exponentially. Moreover, with the adoption of newand more bandwidth-intensive applications, enormous burdens are placedon network infrastructure. Network administrators are thus constantlyseeking faster and more reliable methods and equipment to transport datato accommodate these demands.

Ethernet, one of the earliest networking protocols, is today the mostwidely used method to transport data in computer networks. RobertMetcalf and David Boggs developed Ethernet as an experiment at the XEROXPalo Alto Research Center in 1973. At Ethernet's inception, the struggleto accommodate users needs for bandwidth had not yet started. As networktraffic demands at this time were quite low, Ethernet initially had adata transmission rate of 2.94 megabits per second (Mbps).

Metcalf, however, recognized the potential for rapid network growth andposited a theorem now known as “Metcalf's Law” which states that thevalue of a network expands exponentially as the number of usersincrease. Gordon Moore, an expert in the field of semi-conductordevelopment, posited another theorem known as Moore's Law which statesthat the power of microprocessors will double every 18 months and theirprice will be reduced by half. When taken together, these two lawspredict rapid growth of networking technologies: as users join thenetwork, more people will want to join at an exponential rate equivalentto the rise in value of the network, while processing technologies tosupport this growth and faster transport are constantly increasing atrapidly diminishing costs.

The evolution of Ethernet technologies has followed theory. The firstcommercial release of Ethernet occurred in 1979 with a transmission rateof 10 Mbps—more than a three-fold increase over the experimental systemcreated just five years earlier. Ethernet went through a variety ofstandardizations during the 1980s and line rates remained constant at 10Mbps while the technology matured. In 1995, however, Ethernet becameavailable at 100 Mbps. In 1998, bandwidth jumped again to 1 gigabit persecond (Gbps). Most recently, a new standard was adopted for Ethernettransmission rates at 10 Gbps representing a 100-fold increase in sevenyears.

Implementation of 10 Gbps network infrastructure requires overcomingsignificant hurdles not addressed by current advances in the art. Forexample, previous generations of Ethernet technology, although fast, hadan ample number of clocks in which to perform packet analysis andretransmit data. With the rise of 10 Gbps Ethernet, however,calculations previously carried out over a given number of clocks mustnow be completed in a fraction of the time so that the desired bandwidthis in fact available.

There is thus a need for a systems and methods capable of efficientlyaccommodating data transfer rates over a network in excess of 10 Gbps.

BRIEF SUMMARY OF THE INVENTION

The present invention provides a switch or router for providing datatransmission speeds up to 10 gigabits per second between one or moresource devices and one or more destination devices. The switch includesa blade or board having several discrete integrated circuits embeddedthereon, each performing one or more discrete functions required to meetthe speed required for the switch. The blade includes a media accesscontrol interface (MAC) to facilitate receipt and transmission ofpackets over a physical interface. In one embodiment, the blade furtherincludes four field programmable gate arrays. A first field programmablegate array is coupled to the MAC array and operative to receive packetsfrom the MAC interface and configured to perform initial processing ofpackets. The first field programmable gate array is further operative todispatch packets to a first memory, such as a dualport memory.

A second field programmable gate array is operative to retrieve packetsfrom the first memory and configured to compute an appropriatedestination and to dispatch packets to a backplane. A third fieldprogrammable gate array is operative to receive packets from thebackplane and configured to organize the packets for transmission and todispatch packets to a second memory. A fourth field programmable gatearray is coupled to the MAC interface and operative to retrieve packetsfrom the second memory and to schedule the transmission of packets tothe MAC interface for transmission to one or more destination devices.

According to an alternative embodiment, the invention comprises a switchor router for providing data transmission speeds up to 10 gigabits persecond between one or more source devices and one or more destinationdevices through the use of two sets of one or more field programmablegate arrays. A first set of one or more field programmable gate arraysis coupled to a media access control (MAC) interface and a memorystructure, the MAC interface used to facilitate the receipt andtransmission of packets over a physical interface. The first fieldprogrammable gate array set is operative to receive and transmit packetsfrom and to the MAC interface. The first field programmable gate arrayset is configured to perform initial processing of received packets andto schedule the transmission of packets to the MAC interface fortransmission to one or more destination devices, in addition todispatching and retrieving packets to and from the memory structure.

This embodiment of the invention also comprises a second set of one ormore field programmable gate arrays coupled to the memory structure anda backplane. The second field programmable gate array set is operativeto retrieve packets from and dispatch packets to the memory structure,and configured to compute an appropriate destination and organizepackets for transmission. The second field programmable gate array setis further operative to receive and dispatch packets from and to thebackplane.

BRIEF DESCRIPTION OF THE DRAWINGS

The invention is illustrated in the figures of the accompanying drawingswhich are meant to be exemplary and not limiting, in which likereferences are intended to refer to like or corresponding parts, and inwhich:

FIG. 1 is a block diagram of a system architecture for an Ethernet bladein accordance with one embodiment of the present invention;

FIG. 1A is a block diagram of a system architecture for an Ethernetblade in accordance with a second embodiment of the present invention;

FIG. 2 is a high level flow diagram of a connection of a packetprocessor component of the present invention to an outside network, inaccordance with one embodiment of the present invention;

FIG. 3 is a block diagram of receive and transmit packet processors ofone embodiment of the present invention;

FIG. 4 is a block diagram of a receive packet processor in accordancewith one embodiment of the present invention;

FIG. 5 is a flow diagram showing the data flow in the receive packetprocessor of FIG. 4 in accordance with one embodiment of the presentinvention;

FIG. 6 is a block diagram of a backplane manager in accordance with oneembodiment of the present invention;

FIG. 7 is a flow diagram showing the data flow in a transmissionaccumulator in accordance with one embodiment of the present invention;and

FIG. 8 is a block diagram of a transmit packet processor component inaccordance with one embodiment of the present invention

DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

Embodiments of methods and systems according to the present inventionare described through reference to FIGS. 1 through 8. Turning to FIG. 1,a block diagram is presented depicting a high-level schematic of thecomponents of one possible embodiment of the invention to allow datatransfer speeds at or in excess of 10 gigabits per second. As shown, theinvention comprises a printed circuit board (“PCB”) 10 used to house andprovide interconnections for a media access controller (“MAC”) 12, apacket processor (“PP”) 14, one or more content addressable memory(“CAM”) controllers 16, one or more controllers for random accessmemories containing parameter information (“PRAM”) processors 18, areceive dual-port memory buffer 20, a transmit dual-port memory buffer22, a transmission manager 24, and a backplane interface 26.

The PCB 10 provides a surface on which to place other components of theinvention. The PCB 10, also known as a “blade” or “module”, can beinserted into a slot on the chassis of a network traffic managementdevice such as a switch or a router. This modular design allows forflexible configurations with different combinations of blades in thevarious slots of the device according to differing network topologiesand switching requirements. Furthermore, additional ports for increasednetwork connectivity may easily added by plugging additional blades intofree slots located in the chassis.

An example of such a switch is the BigIron® switch produced by FoundryNetworks, Inc. of San Jose, Calif. The BigIron switch chassis consistsof multiple distributed switching modules each of which contain ahigh-bandwidth memory system for scalable chassis bandwidth. The localswitching fabric of the BigIron switch houses the forwarding engines,provides packet-level examination and classification based on Layer2/3/4 information, and performs IP subnet look-ups and packetmodifications of IP and IPX packets.

The MAC 12 is the interface by which data is received and transmitted toand from the network. In one embodiment, such network data comprisesEthernet packets. The MAC 12 forwards received packets to the PP 14 forfurther processing and also receives packets for transmission to thenetwork from the PP 14. The MAC 12 performs any data conversionsrequired for network data to be processed by the PP 14 for routingwithin the device chassis and for data processed by PP 14 to betransmitted to the network. For example, in one embodiment of theinvention, the MAC 12 performs data conversions because network datacomprises 32 bit double data rate (“DDR”) data, while the PP 14processes only 64 bit single data rate (“SRD”) data. The MAC istypically responsible for data validity checking, as well as datagathering.

The PP 14 is a processor chip responsible for receiving packets from theMAC 12 and processing them for forwarding through the device chassis, aswell as for processing packets received from the device chassis intendedfor transmission over the network. These two functions, while performedon the same chip, are preferably performed simultaneously and inparallel. There are thus, in a sense, two pipelines in the PP 14: areceive pipeline for processing network packets intended fortransmission within the chassis and a transmit pipeline for processinginternally routed packets intended for transmission over the network.

In one embodiment of the invention, the packet processor is a fieldprogrammable gate array (“FPGA”), which is an integrated circuit thatcan be programmed in the field after manufacture. An advantage of usingFPGAs with the invention is that an FPGA provides significantflexibility over an application specific integrated circuit (“ASIC”) andis also much less expensive to prototype and implement.

The receive pipeline of the PP 14 is responsible for packetclassification, performing CAM and PRAM lookups, generating packetheaders for forwarding packets through a chassis, and preparing packetmodifications. Network packets are received by the PP 14 from the MAC 12in multi-byte bursts based on scheduling priorities determined at theMAC 12. The PP 14 examines packets and extracts packet forwardinginformation from the packets such as the destination address (“DA”) ofthe packet and the source address (“SA”) of the packet. The PP 14extracts the type of service (“TOS”), whether the packet has a virtuallocal area network (“VLAN”) tag, session related data such as in thecase of IPv4 or IPX data, and other additional Layer 3 and Layer 4information useful in routing the packet through the chassis. The PP 14passes this forwarding information extracted from the packet header to aCAM processor 16 for further processing.

The CAM controller or processor 16 takes information forwarded by the PP14 and performs a lookup comparing this information to data stored in alocal memory of the CAM processor 16. If the information matchesinformation stored in the local memory of the CAM processor 16,additional forwarding information regarding disposition of the packet isavailable in the local memory of the PRAM processor 18 and can beretrieved for future incorporation into the packet header.

When such successful CAM matches occur, the PRAM processor 18 retrievesadditional forwarding information from its local memory forincorporation into the header of the packet. The packet is reformattedwith a new internal hardware header for routing the packet within thechassis and stored in the receive dual-port memory buffer 20 forprocessing by the transmission manager. This internal hardware header isalso sometimes referred to as a chassis header.

An important technique in implementing the invention is pipelining.Pipelining is an advanced technique used by processors, wherein aprocessor begins executing a subsequent instruction before a priorinstruction has finished executing. Accordingly, a processor can havemultiple instructions processing in its “pipeline” simultaneously witheach instruction at a different processing stage.

The pipeline is divided into processing segments, with each segmentexecuting its operation concurrently with the other segments. When asegment completes its operation, it passes the result to the nextsegment in the pipeline and fetches data for processing from thepreceding segment. Often, temporary memory buffers are used to hold datavalues between segments, which allows operations to complete fastersince each segment no longer waits for the other segment to finishprocessing prior to handing off data. The final results of the processemerge at the end of the pipeline in rapid succession.

The receive dual-port memory 20 (as well as its counterpart, thetransmit dual-port memory 22) acts as a pipeline buffer in theembodiment of the invention depicted in FIG. 1. The receive dual-portmemory 20 enables the PP 14 to store processed data and continueprocessing the next packet without having to wait for the transmissionmanager 24 to become available, thereby expediting operations of boththe PP 14 and the transmission manager 24. Other buffers are usedthroughout the invention and in its various components to achievepipelining and faster packet processing in an analogous manner.

The transmit pipeline of the PP 14 retrieves data from the transmitdual-port memory 22 according to a programmable priority scheme. The PP14 extracts network destinations from the dual-port data and reassemblespacket header forwarding information by removing any packet headermodifications that take place in order to route the packet through theswitch chassis. The PP 14 performs sanity checks on packet data toensure that only those packets intended for transmission are passed onto the MAC 12.

Since packets routed through the chassis carry header informationpertaining to forwarding within the chassis, this information must beremoved and replaced with header forwarding information appropriate forrouting over the network. After the proper network header forwardinginformation is reassembled and the chassis header information isremoved, the PP 14 forwards the data to the MAC 12 for eventualtransmission over the network to the intended address.

While the PP 14 handles traffic to and from the MAC 12 and conversionsof packet headers between network packet headers and internal chassispacket headers, the transmission manager 24 handles traffic flow to andfrom the backplane interface 114. Like the PP 14, the transmissionmanager 24 is a processor chip that implements a dual pipelinearchitecture: a receive pipeline for network data to be internallyrouted within the device chassis and a transmit pipeline for internallyrouted data intended for network transmission. These two functions,while performed on the same chip, are preferably performed in parallelaccording to one embodiment of the invention. In one embodiment of theinvention, the transmission manager 24 is an FPGA, although use of otherprocessor types is within the scope of the invention.

The transmission manager 24 fetches network data intended for routingthrough the device chassis from the receive dual-port memory 20 andstores internally routed data intended for network transmission in thetransmit dual-port memory 22. The receive pipeline of the transmissionmanager 24 retrieves data from the receive dual-port memory 20 accordingto instructions issued to the transmission manager 24 by the PP 14. Thetransmission manager 24 determines data transmission priority for thedata retrieved and schedules transmissions to the backplane 26 accordingto this priority scheme. In one embodiment of the invention, there arefour different priority levels assigned to data.

The transmission manager 24 extracts backplane destinations from data,and sends data to those destinations according to predetermined priorityalgorithms. Backplane destinations may comprise other blades in thechassis or, in some cases, may comprise the blade of the transmissionmanager 24 itself, which is called “one-armed routing.”

The transmit pipeline of the transmission manager 24 handles internallyrouted packets received from the backplane interface 26 and intended fortransmission over the network. The transmission manager 24 collectspackets from the backplane interface 26 and organizes them intoper-source, per-priority transmit queues stored in the transmitdual-port memory 22. The transmission manager 24 notifies the PP 14 whena packet is stored in the transmit dual-port memory 22 and available forprocessing.

FIG. 1 a, presents a block diagram depicting a high-level schematic ofthe components of an alternative embodiment of the invention. As shown,the invention comprises a printed circuit board 100, a media accesscontroller 102, a receive packet processor 104 (“RXPP”), one or more CAMprocessors 106, one or more PRAM memory processors 108, a receivedual-port memory buffer 110, a backplane manager 112, a backplaneinterface 114, a transmission accumulator (“TX accumulator”) 116, atransmit dual-port memory buffer 118, and a transmit packet processor(“TXPP”) 120.

The PCB 100 provides a surface on which to place many of the othercomponents of the invention. The PCB 100, also known as a “blade” or“module”, can be inserted into one of a plurality of slots on thechassis of a network traffic management device such as a switch or arouter. This modular design allows for flexible configurations withdifferent combinations of blades in the various slots of the deviceaccording to differing network topologies and switching requirements.

The MAC 102 is the interface by which a blade receives and transmitsdata to and from the network. In one embodiment, such network datacomprises Ethernet packets. The MAC 102 forwards received packets to theRXPP 104 for further processing and receives packets for transmission tothe network from the TXPP 120. The MAC 102 also performs any dataconversions required for network data to be processed by the RXPP 104 orfor data processed by TXPP 120 to be transmitted to the network. Forexample, the MAC 102 may perform data timing conversions where networkdata comprises 32 bit DDR data while the RXPP 104 and the TXPP 120process only 64 bit SDR data.

The receive packet processor 104 is responsible for packetclassification, performing CAM and PRAM lookups, generating packetheaders for forwarding packets through a chassis, and preparing packetmodifications. In one embodiment of the invention, the receive packetprocessor 104 is an FPGA. In an alternate embodiment of the invention,the RXPP 104 is an ASIC. Packets are received by the RXPP 104 from theMAC 102 in multi-byte bursts based on scheduling priorities determinedat the MAC 102. The RXPP 104 examines packets and extracts packetforwarding information from a packet, such as the destination address ofthe packet and the source address of the packet. The RXPP 104 extractsthe TOS, any defined VLAN tags, session related data such as in the caseof Ipv4 or IPX data, and other additional Layer 3 and Layer 4information useful in routing the packet through the chassis. The RXPP104 passes this forwarding information to one of the CAM processors 106for further examination.

The CAM processor 106 takes information forwarded by the RXPP 104 andperforms a lookup, comparing received information to data stored inlocal memory of the CAM processor 106. If the comparison returns amatch, additional forwarding information regarding disposition of thepacket is stored in local memory of one of the PRAM processors 108 andcan be retrieved for future incorporation into the packet header. ThePRAM processor 108 retrieves additional forwarding information from itslocal memory for incorporation into the header of packet. The packet isthen stored in the receive dual-port memory buffer 110 for processing bythe backplane manager 112. Those of skill in the art will recognize thatadditional processing may be performed before storage in the receivedual port memory.

The receive dual-port memory 110 (as well as its counterpart, thetransmit dual-port memory 118) acts as a pipeline buffer betweenprocesses. The receive dual-port memory 110 enables the RXPP 104 tostore processed data and continue processing the next packet withouthaving to wait for the backplane manager 112 to become available.Pipelining operation execution expedites processing of both the RXPP 104and the backplane manager 112. Other buffers are used throughout theinvention and within its various components to achieve pipelining andfaster packet processing in this manner.

The next segment in the receive pipeline is the backplane manager 112.The backplane manager 112 is a processor designed for retrieving datafrom the receive dual-port memory buffer 110 and dispatching packets tothe backplane interface 114. Data is retrieved from the receivedual-port memory 110 according to instructions issued to the backplanemanager 112 by the RXPP 104. The backplane manager 112 determines datatransmission priority for the data retrieved and schedules transmissionsto the backplane 114 according to this priority scheme. According to oneembodiment of the invention, there are four different priority levelsassigned to data.

The backplane manager 112 extracts backplane destinations from receiveddata; the data sent to indicated destinations according to programmablepriority algorithms. Backplane destinations may comprise other blades inthe chassis or, in the case of OAR, may comprise the blade of thebackplane manager 112 that initially receives the data. When packetsscheduled for OAR are detected, they are forwarded to the transmissionaccumulator 116 via the OAR data path as shown in FIG. 1 a. In oneembodiment of the invention, the backplane manager 112 is an FPGA. In analternate embodiment of the invention, the backplane manager 112 is anASIC.

The transmit accumulator 116 is a processor that receives packet datafrom the backplane 114 intended for transmission. The transmitaccumulator 116 collects packets from the backplane 114 and organizesthem into per-backplane-source, per-priority transmit queues stored inthe transmit dual-port memory 118. The transmit accumulator 116 notifiesthe TXPP 120 when data comprising a packet is stored in the transmitdual-port memory 118 and available for processing. In one embodiment ofthe invention, the transmit accumulator 116 is an FPGA.

The transmit packet processor 120 retrieves data from the transmitdual-port memory 118 according to a programmable priority scheme. TheTXPP 120 extracts network destinations from the data and reassemblespacket header forwarding information by removing any packet headermodifications that took place in order to route the packet through thedevice chassis. The TXPP 120 performs sanity checks on packet data toensure that only those packets intended for transmission are passed onto the MAC 102. Since packets routed through the chassis carry headerinformation pertaining to forwarding within the chassis, thisinformation must be removed and replaced with header forwardinginformation appropriate for routing over the network. After the propernetwork header forwarding information is reassembled and the chassisheader information is removed, the transmit packet processor 120forwards the data to the MAC 102 for eventual transmission over thenetwork to the intended address. In one embodiment of the invention, thetransmit packet processor 120 is an FPGA. In an alternate embodiment ofthe invention, the transmit packet processor 120 is an ASIC.

FIG. 2 presents a high-level schematic of one embodiment of theinvention as it connects to a network, e.g., an optical networkcomprising fiber optic connections. The optics block 202 is theinterface through which all network data traffic passes. The opticsblock 202 contains a transmitter for generating the optical signals tothe network when data is received from the transceiver 204. In someembodiments, the transmitter might comprise a laser or a light emittingdiode. The optics block 202 also contains a detector for receivingoptical data traffic from the network. When optical data is received, aphotodetector generates an electrical current that is amplified to leveluseable by the transceiver 204. The signal is then communicated to thetransceiver 204 for further processing.

The transceiver 204 directs the transmission and receipt of signals toand from the optics block 202. The transceiver 204 receives electricaldata signals intended for transmission to the MAC 206 and instructs thetransmitter in the optics block 202 to generate optical signalscorresponding to the electrical data signals. Conversely, thetransceiver 204 receives electrical data signals from the optics block202 and passes these signals to the MAC 206 for processing.

There are many asynchronous boundaries between the various components ofthe invention. For example, data passes to and from the transceiver 204and the MAC 206 at a fixed speed. In one embodiment of the invention,the datapath 208 between the transceiver and the MAC 206 operatessending 4 clock signals along with 32 bit DDR data at 156.25 MHz. Thedatapath 212 between the MAC 206 and the packet processor 210, however,may operate at a different speed. For example, in one embodiment of thepresent invention, the datapath 212 between the MAC 206 and the packetprocessor 210 operates sending 4 clock signals along with 64-bit SDR at66 MHz. Multiple clock signals are sent with the data and used tominimize timing differences between groups of data signals and a clock.In one embodiment of the invention, one clock signal is included per 8bits of DDR data and one clock signal is included per 16 bits of SDRdata. In addition to clock signals, control signals are also sent alongwith data to indicate packet boundaries and possible error conditions.In one embodiment of the invention, control signals are distributedacross 4 clock groups of data.

Those skilled in the art will recognize that an important technique inmanaging the dataflow between these asynchronous boundaries is the useof FIFO buffers that permit the dataflow to remain synchronized. Giventhe extremely high rate of data transfer provided by the invention,conventional techniques for clock distribution, such as those known inthe art and used in the case of personal computer boards, will not allowreliable capture and transfer of data between components of theinvention operating according to different clocks. The invention,therefore, implements source synchronous clocking wherein the clock issent along with the data.

When the clock arrives at the packet processor 210 from the MAC 206, forexample, the clock is exactly in relationship according to the MAC 206,but the packet processor 210 can also capture the data on that clock viaa FIFO. Data from the MAC 206 is captured inside a FIFO, which allowsthe packet processor to synchronize, in the presence of this data,between the source synchronous clock contained in the FIFO data and theclock the packet processor 210 is using at its core.

The invention uses source synchronous clocking in a symmetric manner.For example, data passing from the packet processor 210 to the MAC 206is also captured in a FIFO to allow the MAC 206 to synchronize, in thepresence of the FIFO data, between the source synchronous clock (of thepacket processor 210 core) and the clock the MAC 206 is using at itscore clock. In an alternative embodiment, the invention also implementsdifferential source synchronous clocking which is known to those skilledin the art. Differential source synchronous clocking works in much thesame manner as source synchronous clocking, except that two clocksignals are sent with the data instead of one clock signal. The twoclock signals, a high and low signal, are used to calculate a moreprecise approximation of the signal value being transmitted which thoseskilled in the art will recognize is used to reduce noise and generatemore accurate data transmissions.

FIG. 3 is a block diagram depicting one embodiment of the components ofthe MAC 102 as presented in FIGS. 1 and 1 a. Components of the MAC 102are embodied in the MAC processor chip 302. According to one embodimentof the invention, the MAC chip 302 is an FPGA. In an alternateembodiment of the invention, the MAC chip 302 is an ASIC. The MAC 102 isthe interface between the network via the PHY transceiver 300 and theRXPP 104 and TXPP 120 packet processor chips. According to oneembodiment of the invention, the MAC 102 communicates directly to thePHY layer transceiver 300 via a DDR interface and with the packetprocessor chips of the RXPP 104 and the TXPP 120 via an SDR interface.

The PHY transceiver 300 is the component applying signals to the networkwire and detecting signals passing through the network wire. Accordingto one preferred embodiment of the invention, the PHY transceiver 300 isa 10 Gigabit Ethernet transceiver transmitting and receiving 32 bit DDRdata at 156.25 Mhz. Data received by the PHY transceiver 300 is passedto the receive front end 306 of the MAC 102. The receive front end 306is an interface that receives data, which is passed to the receive block304 for further processing. According to one preferred embodiment of theinvention, the receive front end 306 receives 32 bit DDR data.

The receive block 304 performs a variety of tasks on data received fromthe receive front end 306 and is very flexible in operation. The receiveblock 304 internally converts data received from the receive front end306 into a format suitable for transmission to the RXPP 104. Accordingto one embodiment of the invention, the receive block converts 32 bitDDR data into 64 bit SDR data for transmission. The receive block 304may also perform other tasks as required according to variousembodiments of the invention such as verifying and extracting XGMIItokens, realigning bytes such that the start of packet (“SOP”) token isplaced in a “lane zero” position, verifying SOP and EOP framing,detecting giant packets, verifying and optionally stripping packetcyclic redundancy checks, tracking the full suite of RMON statistics,and other useful operations.

The receive block 304 also generates flow control packets via the pauseand flow control sync block 332. The receive block 304 operates off ofthe recovered source synchronous clocks contained in the incoming datapackets received from the PHY transceiver 300. Other components of theMAC 102, including the transmit block 328, however, are operating off ofan internal core clock generated locally. Although these two clocks arenominally the same frequency, there is some variance since they are notreally the same clock and therefore tend to “drift” over time. Thisdifference between the two clocks requires periodic synchronization ofthe receive block 304 and the transmit block 328 for the purposes ofpassing flow control messages to generate pause frames and avoid networkcongestion.

In such a scenario, the receive block 304 receives an incoming messagefrom a remote source (to which the transmit block 328 is sending data)indicating that the remote source is becoming congested and requestingthat the transmit block 328 pause transmission for a requested interval.The pause and flow control sync block 332 synchronizes the receive block304 clock with the transmit block 328 clock to permit the receive block304 to pass the pause frame request to the transmit block 328 and reducethe network congestion. Conversely, in the unlikely event that thereceive FIFO RAM 308 becomes congested, the pause and flow control syncblock 332 would synchronize the two clocks to permit the receive block304 to instruct the transmit block 328 to start issuing flow controlpause frames to a remote sender to reduce network congestion in the MAC102.

The receive block 304 passes processed data to the receive FIFO RAM 308via the write port 310 of the receive FIFO RAM 308 which enables thereceive block 304 to process the next packet without waiting for thereceive FIFO block 314 to become available. The receive FIFO RAM 308 isa two-port memory having a write port 310 that accepts incoming datafrom the receive block 304 and a read port 312 that transmits datastored in the receive FIFO RAM 308 to the receive FIFO block 314. Thewrite port 310 and the read port 312 operate independently of each otherthus permitting more efficient use of the receive FIFO RAM 308 by thereceive block 304 and the receive FIFO block 314.

The FIFO RAM 308 further permits data flow though the asynchronousboundary. In one embodiment of the invention, the receive block 304operates at a different speed than the receive FIFO block 314. Thus, theFIFO RAM 308 acts as a bridge, allowing data flow to be synchronizedbetween these asynchronous components. For example, in the FoundryBigIron switch, the receive block 304 operates at a 156.25 MHz clockrecovered from the arriving data and the FIFO block 314 operates on alocally generated 156.25 MHz clock that differs slightly and drifts inphase relationship over time.

To further reduce processing time, the receive block 304 startsstreaming data into the receive FIFO RAM 308 when the receive blockdetects the start of a packet and stops streaming data into the receiveFIFO RAM 308 when the receive block 304 detects the end of the packet.All of the packet processing components of the invention stream datainto FIFOs in this manner which greatly reduces processing time sincecomponents are not required to wait until an entire packet is finishedprocessing to start copying the packet into a FIFO.

The receive FIFO block 314 reads data stored in the receive FIFO RAM 308via the read port 312. The receive FIFO block 314 also notifies the RXPP104 that packet data is contained in the receive FIFO RAM 308 andavailable for transmission. This data is transmitted to the RXPP 104 forfurther processing. According to one embodiment of the invention, thereceive block FIFO 314 transmits 64 bit SDR data to the RXPP 104.

In addition to the receive pipeline of the MAC 102 as set forth above,the MAC 102 also contains a transmit pipeline that operates in a similarfashion with similar flexibility. The transmit FIFO block 320 is theinterface of the MAC 102 that receives data from the TXPP 120. Accordingto one embodiment of the invention, the transmit FIFO block 320 receives64 bit SDR data from the TXPP 120.

The transmit FIFO block 320 streams received data to the transmit FIFORAM 322 via the write port 324 of the transmit FIFO RAM 322, enablingthe transmit FIFO block 320 to process the next incoming packet withoutwaiting for the transmit block 328 to become available. The transmitFIFO RAM 322 is a two-port memory having a write port 324 that acceptsincoming data from the transmit FIFO block 320 and a read port 326 thattransmits data stored in the transmit FIFO RAM 322 to the transmit block328. Similar to the two-port memory comprising the receive FIFO RAM 308,the write port 324 and the read port 326 of the transmit FIFO RAM 322operate independently of each other, thus permitting pipelining and moreefficient use of the transmit FIFO RAM 322 by the transmit FIFO block320 and the transmit block 328.

The transmit block 328 reads data stored in the transmit FIFO RAM 322via the read port 326. Similar to the receive block 304, the transmitblock 328 performs a variety of tasks and is very flexible in operation.The transmit block 328 internally converts data received from TXPP 120into a format suitable for transmission to the PHY transceiver 300.According to one embodiment of the invention, the transmit blockconverts 64 bit SDR data into 32 bit DDR data for transmission. Thetransmit FIFO RAM 322 facilitates this conversion by bridging theasynchronous boundary between the transmit block 328 and the transmitFIFO block 320.

The transmit block performs other tasks as required according toembodiments of the invention, such as generating flow control packets tothe PHY side sender at the request of the TXPP 120 (and in addition tointernal flow control requests generated by the receive block 304 viathe pause and flow control sync 332 when the receive FIFO RAM 308 isfull) to avoid network congestion, calculating and optionally appendinga cyclic redundancy check to a packet, determining and inserting XGMIItokens, and tracking the full suite of RMON statistics. In oneembodiment of the invention, the transmit block 328 stores data in aprogrammable FIFO buffer used for data rate matching which allows theMAC 102 to connect to a packet processor that is receiving data slowerthan line rate.

The transmit block 328 passes data processed for to the transmit frontend 330 thus enabling the transmit block 328 to begin processing thenext packet. The transmit front end 330 is an interface that receivesdata from the transmit block 328 and passes this data to the PHYtransceiver 300 for transmission over the network. According to onepreferred embodiment of the invention, the transmit front end 330transmits 32 bit DDR data to the PHY transceiver 300.

Building on the illustration presented in FIG. 1, FIG. 4 presents ablock diagram depicting one embodiment of the components of the RXPP104. The RXPP 402 is responsible for packet classification, performingCAM and PRAM lookups, generating hardware packet headers used forinternally forwarding packets within the chassis of the network device,such as between blades, and for preparing packet modifications.Components of the RXPP 104 are embodied in the RXPP chip 402. Accordingto one preferred embodiment of the invention, the RXPP chip 402comprises an FPGA. In an alternate embodiment of the invention, the RXPPchip 402 comprises an ASIC.

The XGMAC 404 interface is responsible for requesting data for the RXPP402 from the MAC 102. When the receive lookup handler 406 is availableto parse additional data and the receive data FIFO 438 is available tostore additional data, the XGMAC 404 instructs the MAC 102 to beginstreaming packet data into the RXPP 104. The XGMAC interface 404 isconnected to the MAC 102 and receives data for processing by the RXPP104. The XGMAC interface 404 also acts as an asynchronous boundary,writing source-synchronous 64-bit data from the MAC 102 in a smallinternal FIFO, then sending the synchronized data at 66 MHz in 256-bitchunks for subsequent processing.

The XGMAC interface 404 sends synchronized data as it is received fromthe MAC 102 to the receive data FIFO 438, where it is held until CAM andPRAM lookups are performed. The receive data FIFO 438 thus acts as adelay buffer until packet processing is completed and the packet datacan start being written by the dual-port interface 440 into the receivedual-port memory 110.

While all data related to a packet is streamed to the receive data FIFO438, the XGMAC interface 404 also parses the incoming data as it isreceived from the MAC 102 and streams only the packet header informationto the receive lookup handler 406 where it will be used to perform CAMand PRAM lookups.

The receive lookup handler 406 performs sanity checks on the packet dataas it is received from the XGMAC interface 404. For example, the receivelookup handler 406 identifies valid packet contexts by identifyingconsistent start-of-packet and end-of-packet boundaries. In thisrespect, the receive lookup handler 406 also monitors a bad packetcontrol signal from the MAC 102 indicating a data fault. If a data faultis detected, the receive lookup handler 406 discards the header datafrom the bad packet and also flushes any associated data already storedin the receive data FIFO 438 related to the bad packet. In oneembodiment of the invention, if packet processing has already started, adata fault flag indicating a bad packet is stored in the receive dataFIFO 438. The dual port interface 440 will later discard the packet whenthe data fault flag is retrieved from the receive data FIFO 438.

The receive lookup handler 406 strips VLAN tags, compares the packet MACdestination address against the port MAC address, performs IPv4 TOSfield lookups as required, and also checks the protocol used to encodethe packet. Examples of encoding protocols include IP, IP ARP, IPv4,IPv6, 802.3, IPX RAW, IPX LLC, IPX 8137, IPX SNAP, Appletalk, AppletalkARP, NetBios, IP SNAP, and IP ARP SNAP. This information will be used toassemble an internal hardware packet header to be appended to the packetfor use in forwarding the data internally throughout the chassis of thenetwork switch. This additional information is passed from the receivelookup handler 406 to the RX scheduler FIFO 407. The RX scheduler FIFO407 holds this information until the CAM and PRAM lookups are completedon the destination and source addresses extracted by the receive lookuphandler 406 from the packet header.

Based upon the information extracted, the receive lookup handler 406forms the CAM lookups and builds part of the hardware packet header forinternally forwarding the packet through the chassis of the networkdevice. The internal state of the receive lookup handler 406 containingthis information is then split into two CAM lookup FIFOs 408 and 410,which are memory buffers that permit the receive lookup handler 406 tostart processing the next packet received from the XGMAC interface 404.Packet processing is thus pipelined, allowing the receive lookupprocessor 406 to continue processing packets without waiting for eitherthe CAM1 interface 412 or the CAM2 interface 410 to become available.Information relating to the destination address of the packet and otherprotocol fields from the header related to Layer 3 are passed to CAM1lookup FIFO 408. Information relating to the source address of thepacket and other protocol fields from the header related to Layer 4 arepassed to CAM2 lookup FIFO 410. In an alternate embodiment of theinvention, the two pipelines are merged into a single pipelinecontaining a single CAM interface and a single FIFO interface forlookups.

The CAM1 interface 412 becomes available, retrieves the data stored inthe CAM1 lookup FIFO 408, and submits requests regarding this data tothe external ternary CAM1 414 memory bank that contains a data array ofvalues against which to perform lookups. The CAM1 interface 412 is alsopipelined and supports dispatching lookups for multiple packets to theexternal ternary CAM1 414 memory bank since it takes longer than fourclocks for the external CAM1 414 to respond.

If the lookup generates a match against an entry in the CAM1 414 array,additional forwarding information exists in the PRAM1 426 memory bankregarding the disposition of the packet. Forwarding information mightinclude details such as the destination port of the packet, the portmirror requirement, the packet type, VLAN handling information, packetprioritization data, multicast group membership, replacement destinationMAC addresses (used in network routing), and/or other similar packetdata known in the art. The CAM1 414 array entry also contains a link tothe memory address of the additional forwarding information stored inthe PRAM1 426 memory bank. This link is stored in the CAM1 result FIFO420 until the PRAM1 interface 424 is available to perform lookups.

Similarly, the CAM2 interface 416 retrieves source address data from theCAM2 lookup FIFO 410, performs lookups by submitting requests to theexternal ternary CAM2 memory bank 418, and stores the results of theselookups in the CAM2 result FIFO 422 until the PRAM2 interface 428 isavailable to perform lookups. According to one embodiment of theinvention, the CAM2 interface 416 operates in parallel with the CAM1interface 412 to allow CAM lookup operations to complete faster.

The PRAM1 interface 424 retrieves the data associated with thesuccessful CAM1 interface 412 lookups from the CAM1 result FIFO 420. ThePRAM1 interface 424 extracts from this data the link to the memoryaddress of the additional forwarding information stored in the PRAM1 426memory bank. PRAM1 interface 424 lookup results are stored in the PRAM1result FIFO so work can immediately start on the next packet. Accordingto one embodiment, PRAM lookups for a packet take 3 clocks. Similarly,and preferably in parallel, the PRAM2 interface 428 retrieves dataassociated with successful CAM2 interface 416 source address lookupsfrom the CAM2 result FIFO 422, performs lookups to obtain additionalforwarding information stored in the PRAM2 430 memory bank, and storesthe results in the PRAM2 result FIFO 434.

The receive packet evaluator 436 extracts the data from the PRAM1 resultFIFO 432, PRAM2 result FIFO 434, and the RX scheduler FIFO 407. Thereceive packet evaluator 436 uses this information to construct theinternal hardware header used to forward a packet through the chassiswith the most advanced forwarding in this aspect permitting totaldestination address/VLAN/TOS replacement and packet header modificationto support hardware packet routing. In one embodiment of the invention,the internal hardware header comprises sixteen bytes. The receive packetevaluator 436 also determines the priority level of the packet accordingto the CAM and PRAM lookups and may optionally adjust the packetpriority according to whether the packet is VLAN tagged or contains IPv4TOS fields. The priority level is inserted into the internal hardwareheader of the packet.

The receive packet evaluator 436 notifies the dual-port interface 440that processing is complete and passes the new internal hardware headerto the dual-port interface 440 for integration with the packet datastored in the receive data FIFO 438. The dual-port interface 440 readsfrom the receive data FIFO 438, applying packet modifications toincorporate the new hardware packet header and stores this packet datain the receive dual-port memory 110. The dual-port interface 440 alsodetects the end of packet (“EOP”) signal and issues a receive packetprocessing completion notification to the backplane manager 112 so thebackplane manager 112 will know to retrieve the packet. If a packet isflagged as bad (for example, an invalid cyclic redundancy check) thebuffer is instead immediately recycled for the next packet and thecurrent packet is deleted.

FIG. 5 presents a block diagram depicting the operations of the RXPP 402presented in FIG. 4 more discretely. Data flow commences with thereceive lookup handler 501 receiving packet data from the XGMACinterface 404 as illustrated in FIG. 4. The XGMAC interface 404 parsesdata received from the MAC 102 and sends only the packet headerinformation to the receive lookup handler 501.

The receive port tracker 502 examines the port information contained inthe packet header to ensure that any VLAN information tags contained inthe packet header will be accepted at the destination address port. Ifthe destination address port is not configured to accept the packetheader VLAN information or lack thereof, then the receive lookup handler501 either sets an error bit in the packet header if debugging issupported or the packet is discarded. Alternatively, the receive lookuphandler 501 will strip the VLAN tag from its field in the packet andstore the VLAN tag in the internal hardware packet header for futureuse.

The receive lookup handler 501 checks the protocol used to encode thepacket and classifies the packet accordingly in block 504. Examples ofencoding protocols include IP, IP ARP, IPv4, IPv6, 802.3, IPX RAW, IPXLLC, IPX 8137, IPX SNAP, Appletalk, Appletalk ARP, NetBios, IP SNAP, andIP ARP SNAP. This information is used to assemble an internal hardwarepacket header to be appended to the packet for use in forwarding thedata internally throughout the chassis of the switch. This additionalinformation is passed from the receive lookup handler 501 to the RXscheduler FIFO 522. The RX scheduler FIFO 522 holds this informationuntil the CAM and PRAM lookups are completed on the destination andsource addresses extracted by the receive lookup handler 501 from thepacket header.

The receive lookup handler 501 also forms the CAM lookups and buildspart of the hardware packet header in block 506. The receive lookuphandler 501 extracts source and destination address information from thepacket header for use in the CAM lookups. The internal state of thereceive lookup processor 501 containing this information is then passedto the CAM lookup FIFO 508, which is a memory buffer that permits thereceive lookup processor 501 to start processing the next packetreceived from the XGMAC interface 404. Packet processing is thuspipelined allowing the receive lookup processor 501 to continueefficiently processing packets without waiting for the CAM interface 509to become available.

When the CAM interface 509 becomes available, it fetches the addressdata stored in the CAM lookup FIFO 508 as shown in block 510. The CAMinterface 509 dispatches requests regarding data in block 512 to theexternal ternary CAM memory 516 that contains a data array of valuesagainst which to perform lookups. The CAM interface 509 is pipelined andsupports cycling lookups for multiple packets to the external ternaryCAM 516 memory since it takes longer than four clocks for the externalCAM 516 to respond. Block 514 illustrates a programmable delayincorporated into the CAM interface 509 pipeline that compensates forthis delay while the CAM lookup is being performed.

If the lookup generates a match against an entry in the CAM array 516,additional forwarding information regarding disposition of the packet isavailable in the PRAM memory 530. Forwarding information might includedetails such as the destination port of the packet, the port mirrorrequirement, the packet type, VLAN handling information, packetprioritization data, multicast group membership, and/or other similarpacket data known in the art. The CAM array 516 entry also contains alink to the memory address of the additional forwarding informationstored in the PRAM memory 530. This link is returned by the CAM memory516 as shown in block 518 and stored in the CAM result FIFO 520 untilthe PRAM interface 523 is available to perform lookups.

When the PRAM interface 523 becomes available, it fetches the link tothe address in the PRAM memory 530 that is stored in the PRAM lookupFIFO 520 as shown in block 524. In block 526, the PRAM interface 523dispatches requests to retrieve the additional forwarding informationfor the packet to the external PRAM memory 530. The PRAM interface 523is pipelined and supports cycling lookups for multiple packets to theexternal PRAM memory 530 since it takes multiple clocks for the externalPRAM memory 530 to return results from a lookup. Block 528 illustrates aprogrammable delay incorporated into the PRAM interface 523 pipelinethat compensates for this delay while the PRAM lookup is beingperformed. The external PRAM 530 returns the additional forwardinginformation in block 532 and these results are stored in the PRAM resultFIFO 534 until the receive packet evaluator 535 is available.

In block 536, the receive packet evaluator 535 fetches data from thePRAM result FIFO 534 and the receive scheduler FIFO 522. The receivepacket evaluator 535 evaluates this information in block 538 and usesthe results to construct the internal hardware packet header in block540. The internal hardware packet header is used to forward the packetthrough the chassis among other blades inserted into slots on thebackplane. The most advanced forwarding in this aspect permits totaldestination address/VLAN/TOS replacement and packet header modificationto support hardware packet routing. In one embodiment of the invention,the internal hardware header comprises sixteen bytes.

The receive packet evaluator 535 notifies the dual-port interface 542that processing is complete and passes the new internal hardware headerto the dual-port interface 542 for integration with the packet datastored in the receive data FIFO 438, as illustrated in FIG. 4. Thedual-port interface 542 reads from the receive data FIFO 438 applyingpacket modifications to incorporate the new hardware packet header forinternally forwarding the packet through the chassis of the switch andstores this packet data in the receive dual-port memory 110. The receivedual-port memory is organized as four large FIFOs corresponding to fourexemplary priority levels. The dual-port interface 440 also detects theend of packet (“EOP”) and issues a receive packet processing completionnotification to the backplane manager 112 so the backplane manager 112will know to retrieve the packet. If a packet is flagged as bad (forexample, an invalid cyclic redundancy check) the packet is deleted andthe buffer is recycled for the next packet.

Transport within a blade continues with FIG. 6, which presents a blockdiagram depicting the components of the backplane manager 112 asillustrated in FIG. 1. Components of the backplane manager 602 areembodied in the backplane manager chip. According to a embodiment of theinvention, the backplane manager chip 602 comprises an FPGA.

The backplane manager 602 is responsible for retrieving data from thereceive dual-port memory 610, determining backplane destinations forthis data, and sending this data to those destinations. The backplanemanager 112 also manages four large FIFOs stored in the externaldual-port memory 610. These FIFOs store data according to prioritylevels by which the data is to be processed by the backplane manager112.

The receive done handler 604 receives EOP information from the receivepacket processor 104, including information regarding packet length andpacket priority. This information is used to assist the receive donehandler 604 in tracking receive dual-port memory 110 utilization for thefour priority levels and scheduling packets for dispatch by the transmitqueue dispatch 606. If the backplane manager 602 or the receivedual-port memory FIFOs 610 are running low on resources, the receivedone handler 604 sends a throttle control back to the receive packetprocessor 104.

The transmit queue dispatch 606 is responsible for ordered packetdispatch from the four priority levels of the receive dual-port memoryFIFOs 610. The transmit queue dispatch 606 receives packet length andpriority information from the receive done handler 606 and uses thisinformation to schedule packet retrieval from the dual-port RAM 610 bythe dual-port interface 608 according to prioritization algorithmscontained in the transmit queue dispatch 606.

According to one embodiment of the invention, absolute priority is usedwith higher priority packets being unconditionally transmitted beforeany packets of lower priority. Absolute priority, however, is not alwaysdesirable. In another embodiment, some fraction of the transmissionbandwidth available to the backplane manager 112 is dedicated to lowerpriority packet transmission regardless of whether higher prioritypackets are also pending because packets are often received by theinvention faster than they can be transmitted. If some bandwidth werenot allocated to lower priority packets in this manner, a bottleneckmight be created with lower priority packets not being transmitted dueto higher priority packets monopolizing all available transmissionbandwidth. Packets are thus scheduled and posted for use by the transmitqueue dispatch 606.

The dual-port interface 608 fetches data from the receive dual-portmemory 610 based on instructions received by the transmit queue dispatch606. At the start-of-packet boundary, the dual-port interface 608extracts a forwarding identifier (“FID”) from the packet and sends theFID to the FID lookup interface 612. The FID is an abstractchassis/system wide number used to forward packets. Each packet type hasa FID to instruct the blade how to handle a given type of packet. Thisallows each blade in the chassis to look at the FID separately to decidehow to individually forward the packet.

The FID lookup interface 612 translates the FID received from thedual-port interface 608 into a port mask by performing a lookup againstaddresses stored in the external FID RAM 614. The port mask is amulti-bit field representing a port on the blade and also other possiblebackplane slot destinations in the device chassis. According to oneembodiment, the port mask is an 8-bit field representing a 10 GigabitEthernet port on the blade and seven other possible backplane slotdestinations.

The FID lookup takes a number of clock cycles to complete during whichtime read data is posted to the delay FIFO 616 by the dual-portinterface 608. According to one embodiment of the invention, the FIDlookup by the FID lookup interface 612 into the external FID RAM 614requires a delay of six clocks to complete in order to resume processingthe data.

The FID lookup is completes and the results are passed from the FIDlookup interface 612 to the merge port mask 618. Read data stored in thedelay FIFO 616 is also passed to the merge port mask 618. The merge portmask 618 integrates the read data with the appropriate FID lookup portmask result and other port masks as set forth below to ensure that thedata is transmitted to all intended destinations.

The merge port mask 618 takes the FID lookup port mask result andcombines it with CPU and monitor information stored in configurationregisters of the backplane manager. For example, a FID indicates aphysical destination or possibly a list of destinations, but the receivepacket processor 104 might have determined that the CPU also needs acopy of the data and therefore sets the CPU flag for combination withthe FID lookup port mask by the merge port mask 618. Alternatively, whena packet needs to be sent to a monitor port for network debugging orsimilar purpose, the monitor port mask is combined with the FID portmask. The merge port mask 618 thus generates a “qualified” port maskindicating all destinations for which the packet data is intended.

The merge port mask 618 may also apply source port suppression. Incertain situations, the blade that receives the data packet is listed aspart of a FID port mask; source port suppression conditionally preventsthe blade from retransmitting packets it just received. For example,this might occur in a broadcast situation where packets with unknownaddresses are sent to all ports. Once all port mask data is combinedwith packet data, the merge port mask 618 stores the final result in thereceive data FIFO 620 enabling the merge port mask 618 to process thenext packet without waiting for the backplane FIFO dispatch 624 tobecome available.

The backplane FIFO dispatch 624 reads data from the receive data FIFO620, duplicating the data for each destination indicated in thequalified port mask. The backplane FIFO dispatch 624 restructures thedata into a format required by the backplane, generates backplane stateand slot information, and posts the results into the backplane data FIFO626. The backplane data FIFO 626 also acts as an asynchronous boundarybetween the backplane manager 602 core clock and the actual backplaneclock. By posting the results in the backplane data FIFO 626, thebackplane FIFO dispatch 624 can process the next packet without waitingfor the backplane dispatch 628 to become available. In one embodiment ofthe invention, data posted to the backplane data FIFO 626 is equivalentto two backplane transfers since the backplane manager runs atapproximately one-half the clock speed of the backplane interface 114.

The backplane dispatch 628 reads data from the backplane data FIFO 626and outputs the data to the backplane via the backplane interface 114.According to one embodiment, the backplane dispatch 628 reads data fromthe backplane data FIFO 626 suitable for more than one transfer becausethe ratio of the backplane interface 114 clock speed and the clock speedof the backplane manager 602 is not identical. In such an embodiment,the backplane dispatch 628 reads the number of transfers from thebackplane data FIFO 626 that fully utilizes the transmission capacity ofthe backplane interface 114. For example, if the clock speed of thebackplane interface 114 is double that of the backplane manager 602,then the backplane dispatch 628 will read two transfers from thebackplane data FIFO.

The backplane dispatch 628 also monitors backplane status and directsbackplane transmission rates since it is possible for a backplane slotdestination to become congested or otherwise unavailable. For example,if a plurality of blades comprising a single chassis are devoting all oftheir transmission capacities to a single blade, then they may overloadthe destination blade. Such a case might occur when two blades bothtransmit at 8 Gbps to a single destination blade that, according to thecapacity of a backplane slot, can only receive 8 Gbps it total. The twoblades would have to throttle back transmissions to the destinationblade to 4 Gbps to avoid congestion.

Data is received from the backplane by the transmission accumulator 116as presented in FIG. 1. Turning to FIG. 7, the transmission accumulator116 collects packets from the backplane and organizes them intoper-source, per priority transmit FIFOs stored in the transmit dual-portmemory 118. Components of the transmission accumulator are embodied inthe transmission accumulator chip 702. According to one embodiment ofthe invention, the transmission accumulator chip 702 comprises an FPGA.

Data is received from the backplane by the backplane front end 704. Thebackplane front end passes received data to the backplane slot receiveaccumulator 706. The backplane slot receive accumulator 706 is dividedinto a series of equal storage structures or memory buffers, with onebuffer allocated for each slot or source on the chassis of the device.According to one embodiment of the invention, the backplane slot receiveaccumulator 706 is divided into eight buffers for receipt of data.

When a particular quantity of data is received into one of the backplaneslot receive accumulator 706 buffers, the backplane slot receiveaccumulator 706 notifies the backplane data polling logic 708 toindicate the buffer and priority of the data being stored. In oneembodiment of the invention, the backplane slot receive accumulator 706waits to notify the backplane data polling logic 708 until 32 bytes ofdata have been received in a bucket and transfers between the twocomponents thus comprise 32 bytes. If the backplane slot receiveaccumulator 706 is full, then the transmission accumulator is congestedand no longer accepts data until the congestion is relieved.

The backplane data polling logic 708 reads data from the backplane slotreceive accumulator 706 and organizes data according to source andpriority. If packets are aborted from the backplane, the backplane datapolling logic 708 deletes the packet in order to avoid propagation ofthe packet to the TXPP 120.

The backplane data polling logic 708 processes the data and the finalresult is stored in the backplane receive FIFO 710, enabling thebackplane data polling logic 708 to process the next packet withoutwaiting for the dual-port interface 712 to become available. Thebackplane receive FIFO 710 also permits dataflow through theasynchronous boundary between the backplane data polling logic block 708and the dual-port interface 712.

The dual-port interface 712 reads data from the backplane receive FIFO710 and stores this packet data in the transmit dual-port memory 118.The dual-port interface 712 also detects valid end-of-packet (“EOP”)indications and notifies the TXPP 120 via transmission of an EOP messagethat a packet is available in the transmit dual-port memory 118. Thetransmit dual-port memory 118 also comprises a series of FIFOs similarto the receive dual-port memory 110. Instead of only four total FIFOs,however, the transmit dual-port memory 118 has four FIFOs for eachbuffer of the backplane slot accumulator 706, thereby comprising 28FIFOs for these buffers, plus an additional four FIFOs for the OAR path,yielding a total of 32 FIFOs.

Transmission continues in FIG. 8, which depicts a block diagram of thecomponents of the transmit packet processor 120 as illustrated in FIG. 1a. Components of the TXPP 120 are embodied in the TXPP chip 800.According to an embodiment of the invention, the TXPP chip 800 comprisesan FPGA. The TXPP 800 is responsible for retrieving data from thetransmit dual-port memory 803, determining network destinations for thisdata and sending data to identified destinations. The TXPP 120 stripshardware header forwarding information used to route packets throughoutthe chassis of the switch and replaces this information with headerforwarding information necessary to route packets over the network. TheTXPP 120 also manages the FIFOs priority queues stored in the transmitdual-port memory 803. These FIFOs store data according to prioritylevels by which the data is to be processed by the TXPP 800.

The transmit done handler 801 receives EOP information from the TXaccumulator 116, including information regarding packet length andpacket priority. This information is used to assist the transmit donehandler 801 in tracking transmit dual-port memory 803 utilization forthe four priority levels and scheduling packets for dispatch in thetransmit queue dispatch 802. The transmit done handler 801 notifies thetransmit queue dispatch 802 regarding packet availability and priority.

The transmit queue dispatch 802 is responsible for ordered packetretrieval and dispatch from the four priority levels of the transmitdual-port memory 803 FIFOs. According to one embodiment of theinvention, absolute priority is used with higher priority packets beingunconditionally transmitted before any packets of lower priority.Absolute priority, however, is not always desirable. In alternativeembodiments, some fraction of the transmission bandwidth available tothe TXPP 120 is dedicated to lower priority packet transmissionregardless, of whether higher priority packets are also pending becausepackets are often received by the invention faster than they can betransmitted. If some bandwidth were not allocated to lower prioritypackets in this manner, a bottleneck might be created with lowerpriority packets not being transmitted due to higher priority packetsmonopolizing all available transmission bandwidth. Packets are thusscheduled and posted for use by the dual-port handler 804.

The dual-port handler 804 fetches the data from the transmit dual-portmemory 803 according to instructions received from the transmit queuedispatch 802. At the start-of-packet boundary, the dual-port handler 804extracts the FID from the packet and sends the FID to the FID lookupblock 808. The dual-port handler 804 also extracts any VLAN tags fromthe packet and sends this information to the multicast start offsetlookup block 806.

In the FID lookup block 808, the FID received from the dual-port handler804 is used to perform a lookup against a FID table. The FID lookupblock 808 functions similarly to the interaction between the FID lookupinterface 612 and the FID RAM 614 as presented in FIG. 6. Accordingly,the results obtained from the FID table indicate how the packet shouldbe handled for transmission by the receiving blade. For example, the FIDmight indicate that although the packet may have arrived at the blade,the packet should not be transmitted by the blade. This might occur in abroadcast situation where a packet is broadcast to all blades within achassis. If the FID lookup block 808 determines that a packet has beenerroneously received in this manner, the packet is deleted and no longerprocessed by the TXPP 120. In this sense, the FID lookup block 808 alsofunctions as a transmit filter to ensure that only valid packets areactually sent out over the network.

Results of the FID lookup are stored in the delay FIFO 810. This permitsthe FID lookup block 808 to begin processing the next packet withoutwaiting for the context track and internal header removal block 814 tobecome available. Pipelining processing data in this manner allowspacket processing operations by the TXPP 120 to complete faster.

While the FID lookup block 808 is processing the FID data, the multicaststart offset lookup block 806 is processing any VLAN tags received fromthe dual-port handler 804. A VLAN is a local area network identifierthat maps locations based on a basis other than physical location. Forexample, devices attached to a VLAN might be grouped according todepartment, division, application, etc. Devices that are part of thesame VLAN behave as if they were connected to the same wire even thoughthey may actually be physically connected to different segments of aLAN. VLANs are configured using software protocols rather than inhardware and are therefore extremely flexible with respect toimplementation. For example, a computer may be moved to a differentphysical location on the same VLAN without any hardware reconfiguration.

VLAN tags placed in a header field indicate whether a packet is intendedfor routing over a VLAN. Additionally, the VLAN tag in the header mayalso indicate that a packet is intended for VLAN multicasting. VLANmulticasting occurs when a packet is sent over a VLAN to more than onedestination address. Since the header of each packet must be changed toreflect each destination address during VLAN multicasting, this processcan be very resource intensive when performed using software.

The multicast start offset lookup block 806 supports hardware VLANmulticast replication. The multicast start offset lookup block 806examines the VLAN tag extracted from the packet header and performs alookup against a table stored in RAM in the multicast start offsetlookup block 806. If the packet VLAN tag matches an entry in the table,additional information pertaining to that VLAN is available at anaddress location in a memory array stored in the multicast replacementlookup block 812. For example, multicast replacement lookup block 812might contain information to assist with setting unique VLAN ID values,VLAN priorities, and TXA/SAS/srcport suppression behaviors for eachpacket transmitted over the VLAN.

The multicast start offset lookup block 806 takes the address to thememory array location of the multicast replacement lookup block 812 andstores this result in the delay FIFO 810. This permits the multicaststart offset lookup block 806 to begin processing the next packetwithout waiting for the context track and internal header removal block814 to become available. Pipelining processing in this manner allowspacket processing operations by the TXPP 120 to complete faster.

In addition to enabling pipelining, the delay FIFO 810 also storesvalues from the FID lookup block 808 and the multicast start offsetlookup block 806 for retrieval by the multicast replacement lookup block812 and the context track and internal header removal block 814. Themulticast replacement lookup block 812 retrieves the results of themulticast start offset lookup block 806 calculations from the delay FIFO810 for processing packets subject to VLAN routing. The multicastreplacement lookup block 812 takes the address of the memory arraylocation contained in the multicast replacement lookup block 812 andretrieves the additional information that is stored at that locationpertaining to routing over the VLAN tag referenced in the packet header.This information is passed to the context track and internal headerremoval block 814 for incorporation into the outgoing packet header.

Taking the results from the delay FIFO 810 and the multicast replacementlookup block 812, the context track and internal header removal block814 removes the internal hardware header from the packet and begins theprocess of assembling an outgoing packet header suitable fortransmission over the network. Those skilled in the art will recognizethat a number of manipulations to the outgoing packet header must takeplace before this can occur. The context track and internal headerremoval block 814 passes information regarding any data offset to theheader which may have occurred to the barrel shifter 816. The contexttrack and internal header removal block 814 passes information regardingthe TXA/PTYPE to the SA substitution and L3 assist block 818. Thecontext track and internal header removal block 814 passes informationregarding the packet VLAN ID and the VLAN tag status to the VLANinsertion block.

The barrel shifter 816 normalizes any changes to the packet header thatoccurred during internal routing through the chassis. One function ofthe internal hardware header of a packet is to permit the CPU to add anencapsulation to a packet. Encapsulation is used by the CPU to completeoperations more efficiently by avoiding having to copy the entire packetinto CPU memory and then writing the packet back to the buffer pool.Instead, the CPU performs a small modification to the packet header. Forexample, this might occur when the CPU determines that a packet must beforwarded, but that the CPU must first add data to the header beforeforwarding can take place. Alternatively, the CPU might also remove datafrom the header temporarily to assist with forwarding.

During this process, the CPU might move data within the packet headerinto a non-standard format. For example, the destination address mightappear at the wrong location within the packet for transmission over thenetwork. The barrel shifter 816 analyzes the composition of the packetheader and shifts the data within the header to normalize it and correctfor any CPU modifications that might have occurred. When the barrelshifter 816 completes operations on the packet header, the packet headerdata is then in a standard format and is passed to the SA substitutionand L3 assist block 818 for further processing.

The SA substitution and L3 assist block 818 performs furthermodifications on the packet header to prepare the packet fortransmission over the network. The SA substitution and L3 assist block818 replaces the MAC address that is required for routing packets. In anEthernet environment, each packet header contains a destination addressand a source address. The source address must be changed on transmit toreflect which port the packet is being broadcast from. The SAsubstitution and L3 assist block 818 also modifies other Layer 3 headerfields as required, such as changing the IPv4/IPX time to live value orthe checksum.

The packet is passed to the VLAN insertion block 820 for furtherprocessing. VLAN tags that were removed on receipt anywhere in thechassis are stored in the internal hardware header for future use ontransmission. The VLAN insertion block 820 takes the internal hardwareheader information that is passed from the context track and internalheader removal block 814 and reintroduces this information into theoutgoing packet header as appropriate. This information includes thepacket VLAN ID and the Tag Status.

When the outgoing header packet is reassembled for transmission over thenetwork, the packet is stored in the TX FIFO 822 prior to being passedto the XGMAC interface 824. The TX FIFO 822 enables the VLAN insertionblock 820 to begin processing the next packet without having to wait forthe XGMAC interface to become available and enables faster operation bythe VLAN insertion block 820.

Additionally, the TX FIFO 822 permits data flow though asynchronousboundaries. In some embodiments of the invention, the TXPP 120 operatesat a different speed than the MAC 102. Data flow must be synchronizedbetween asynchronous components so the TX FIFO 822 acts as a bridgebetween these components. For example, in the Foundry BigIron switch,the MAC 102 operates at a 156.25 MHz clock and the TXPP operates at onlya 66 MHz clock.

While the invention has been described and illustrated in connectionwith preferred embodiments, many variations and modifications as will beevident to those skilled in this art may be made without departing fromthe spirit and scope of the invention, and the invention is thus not tobe limited to the precise details of methodology or construction setforth above as such variations and modification are intended to beincluded within the scope of the invention.

1. A system to provide data transmission speeds at or in excess of 10gigabits per second between one or more source devices and one or moredestination devices, the system comprising: a media access control (MAC)interface to facilitate receipt and transmission of packets over aphysical interface; a first processor coupled to the MAC interface toreceive packets from the MAC interface and to perform initial processingof packets, and further to dispatch packets to a first memory; a secondprocessor to retrieve packets from the first memory and to compute anappropriate destination, and further to dispatch packets to a backplane;a third processor to receive packets from the backplane and to organizethe packets for transmission, and further to dispatch packets to asecond memory; and a fourth processor coupled to the MAC interface andto retrieve packets from the second memory and to schedule thetransmission of packets to the MAC interface for transmission to one ormore destination devices.
 2. The system of claim 1 wherein the first andsecond memories are dualport memories.
 3. The system of claim 1, furthercomprising a plurality of FIFO structures within each of said processorsto provide for pipelined processing of data.
 4. The system of claim 3wherein said processors are configured to process a packet in fourclocks.
 5. The system of claim 3 wherein the first processor comprises areceive packet processor to transfer data in at least 64 bit groupings.6. The system of claim 5 wherein the receive packet processor isconfigured to perform CAM and PRAM lookups for received packets.
 7. Thesystem of claim 6, further comprising FIFO structures coupled betweenCAM and PRAM lookups to accommodate pipelined processing.
 8. The systemof claim 7 wherein the receive packet processor is configured to performmultiple CAM and PRAM lookups in parallel.
 9. The system of claim 8wherein at least one of the multiple CAM lookups is a source address CAMlookup and at least one other of the multiple CAM lookups is adestination address CAM lookup.
 10. The system of claim 8 wherein atleast one of the multiple PRAM lookups is a source address PRAM lookupand at least one other one of the multiple PRAM lookups is a destinationaddress PRAM lookup.
 11. The system of claim 1 wherein the secondprocessor comprises a backplane manager to issue packets on thebackplane.
 12. The system of claim 11 wherein the backplane manager isconfigured to compute a packet priority.
 13. The system of claim 12wherein the packet priority is computed based on absolute priority. 14.The system of claim 12, further comprising means for providing lowpriority packets a guaranteed fraction of bandwidth.
 15. The system ofclaim 1 wherein the third processor comprises a transmission accumulatorto organize packets into a plurality of per-source and per-priorityFIFOs.
 16. The system of claim 15 wherein the FIFOs are contained in thesecond memory.
 17. The system of claim 15 wherein the transmissionaccumulator is configured to delete aborted packets.
 18. The system ofclaim 17 wherein the backplane manager is configured to issue anend-of-packet message to the fourth processor to indicate that acomplete packet is available for retrieval.
 19. The system of claim 1wherein the fourth processor is a transmission packet processor toimplement a priority scheduler.
 20. The system of claim 19 wherein thepriority scheduler is configured to determine an order in which packetsare transmitted from the transmission packet processor to the MACinterface, based on their priority.